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March  25,  2011 
Congressional  Committees 

Subject:  Defense  Acquisitions:  Application  of  Lessons  Learned  and  Best  Practices 
in  the  Presidential  Helicopter  Program 

In  June  2009,  following  the  expenditure  of  close  to  $3  billion  and  a  critical  Nunn- 
McCurdy  breach^  of  the  cost  growth  threshold,  the  Department  of  Defense  (DOD) 
terminated^  the  Navy’s  VH-71  presidential  helicopter  acquisition  program  and 
contract  because  of  cost  growth,  schedule  delays,  and  projected  system  performance. 
The  Presidential  Helicopter  VXX  program  is  a  successor  Navy  program  to  the 
terminated  VH-71  program  acquisition  and  has  been  initiated  to  develop  aircraft  to 
replace  the  current,  aging  presidential  helicopter  fleet. 

The  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year  201 1  (the  Act) 
directed  GAO  to  review  and  report  annually  to  the  congressional  defense  committees 
on  the  VXX  program  through  2013.®  This  is  the  first  of  the  required  GAO  reports.  It 
discusses  (1)  major  lessons  learned  from  the  terminated  VH-71  program  that  should 
be  applied  to  the  follow-on  VXX  program  and  (2)  the  current  acquisition  approach  of 
the  VXX  program  and  sufficiency  of  the  underlying  acquisition  plans  and  related 
documentation. 

To  identify  major  VH-71  program  lessons  learned,  we  examined  the  cost,  schedule, 
and  performance  experience  of  the  terminated  VH-71  program  through  discussions 
with  agency  officials  and  review  of  reported  lessons  learned  and  then  assessed  how 


'  10  U.S.C.  §  2433  establishes  the  requirement  for  the  Department  of  Defense  (DOD)  to  prepare  unit 
cost  reports  on  major  defense  acquisition  programs  or  designated  major  defense  subprograms.  If  a 
program  exceeds  cost  growth  thresholds  specified  in  the  law,  this  is  commonly  referred  to  as  a  Nurm- 
McCurdy  breach.  DOD  is  required  to  report  these  breaches  to  Congress,  and  in  certain  circumstances, 
DOD  must  reassess  the  program  and  submit  a  certification  to  Congress  in  order  to  continue  the 
program,  in  accordance  with  10  U.S.C.  §  2433a.  The  Nunn-McCurdy  breach  experienced  by  the  VH-71 
program  was  estimated  to  represent  program  acquisition  unit  cost  (PAUC)  growth  of  more  than  50 
percent  above  the  original  acquisition  program  baseline — a  level  that  would  have  required  certification 
to  Congress  for  continuation  of  the  program  if  the  program  had  not  been  terminated.  PAUC  represents 
the  expected  acquisition  cost  for  each  unit  procured,  as  determined  by  dividing  the  sum  of  a  program’s 
total  program  development,  procurement,  and  system-specific  military  construction  costs  by  the 
number  of  fuhy  configured  end  items  to  be  procured. 

^  A  memorandum  canceling  the  program  was  issued  in  May  2009.  Subsequently,  a  notice  terminating 
the  contract  was  sent  to  the  contractor  in  June  2009.  In  accordance  with  the  termination  clause,  the 
contractor  was  required  to  submit  a  termination  settlement  proposal  within  1  year.  The  contractor 
submitted  its  termination  proposal  in  May  2010.  The  termination  proposal  is  currently  being  audited 
and  a  final  settlement  is  expected  by  the  end  of  fiscal  year  2011. 

'Pub.  L.  No.  111-383  §233. 
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those  lessons  relate  to  acquisition  best  practices.  To  determine  the  current  status  of 
the  VXX  program,  we  examined  available  program  documents  and  interviewed 
knowledgeable  officials.  To  assess  the  sufficiency^  of  the  program’s  efforts,  we 
discussed  the  Navy’s  plans  for  the  program  with  knowledgeable  officials  and 
obtained  and  analyzed  the  one  completed  VXX  program  baseline  acquisition 
document,  the  program’s  Initial  Capabilities  Document  (ICD),®  to  determine  whether 
it  incorporates  acquisition  best  practices  and  addresses  DOD’s  acquisition  policy  and 
guidance.  We  also  obtained  preliminary  information  on  the  ongoing  VXX  program 
Analysis  of  Alternatives  (AO A)®  and  made  preliminary  observations  on  this  effort  and 
the  terminated  VH-71  program’s  AOA  conducted  in  2003. 

We  conducted  this  performance  audit  from  September  2010  to  March  2011  in 
accordance  with  generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence 
to  provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objectives.  We  believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 

Background 

The  Marine  Corps’s  HMX-1  squadron  currently  utilizes  two  types  of  helicopters — the 
VH-3D  and  the  VH-60N — to  carryout  the  presidential  helicopter  mission.  Over  the 
past  several  years,  numerous  modifications  and  improvements  have  been  made  to 
both  aircraft  types  to  incorporate  emerging  technologies  and  address  new 
requirements.  These  improvements  have  increased  the  gross  weight  of  the 
helicopters — decreasing  some  aspects  of  mission  capability,  for  example,  range,  and 
severely  limiting  the  ability  to  incorporate  future  improvements  because  of  the 
impact  of  further  weight  growth. 

According  to  program  officials,  over  time  it  had  become  increasingly  difficult  to 
accommodate  the  demands  placed  on  the  HMX-1  aircraft  in  support  of  presidential 
requirements.  The  events  following  the  terrorist  attacks  on  the  United  States  on 
September  11,  2001,  highlighted  the  need  for  improved  transportation, 
communication,  and  security  capabilities  for  presidential  support  aircraft.  As  a  result, 
a  VXX  replacement  helicopter  program  was  initiated  in  April  2002  to  begin  fielding  a 
new  helicopter  in  the  2011  time  frame.  A  subsequent  November  2002  White  House 
memo  to  the  Secretary  of  Defense  articulated  a  goal  of  having  the  new  helicopter 
available  earlier,  and  in  response,  the  Navy  developed  an  accelerated  program  plan  to 
develop  and  initially  field  a  new  helicopter  by  the  end  of  2008.  The  VXX  program  was 
subsequently  redesignated  the  VH-71  program. 


^  We  have  defined  sufficiency  as  meaning  objective  and  in  accord  with  acquisition  best  practices  and 
DOD  acquisition  policy  and  guidance. 

^  A  sponsor,  usually  a  military  service,  submits  a  capability  proposal  called  an  ICD  through  DOD’s 
requirements  determination  process — the  Joint  Capabilities  Integration  and  Development  System.  An 
ICD  identifies  the  existence  of  capability  gap(s),  the  operational  risks  associated  with  the  gap(s),  and  a 
recommended  solution  or  preferred  set  of  solutions  for  filling  the  gap(s). 

An  AOA  is  an  evaluation  of  the  performance,  effectiveness,  suitability,  and  estimated  costs  of 
alternative  systems  to  meet  a  capability. 
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In  January  2005,  the  Navy  entered  into  a  contract  with  Lockheed  Martin  Systems 
Integration  to  develop  the  replacement  helicopter.  By  January  2009,  after  schedule 
slips  and  significant  cost  increases,  the  Navy  reported  a  Nunn-McCurdy  breach  of  the 
critical  cost  growth  threshold  and  in  June  2009  terminated  the  VH-71  program. 
However,  the  need  for  a  replacement  helicopter  remains.  An  ICD  was  drafted  by  the 
Marine  Corps  and  approved  by  DOD  in  August  2009  to  start  a  successor  VXX 
Helicopter  Replacement  Program.  According  to  program  officials,  the  current 
projected  VH-3D  and  VH-60N  service  life  will  support  the  development  and  fielding  of 
the  VXX. 

In  the  last  few  years.  Congress  and  DOD  have  put  in  place  new  defense  acquisition 
policy  provisions  that  replace  risk  with  knowledge — placing  greater  emphasis  on 
front-end  planning  and  establishing  sound  business  cases  for  starting  programs.  For 
example,  both  DOD’s  December  2008  acquisition  policy  revisions  and  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  require  programs  to  invest  more  time  and 
resources  in  the  front  end  of  the  acquisition  process  in  line  with  a  knowledge-based 
acquisition  approach.  ^  As  a  result,  DOD’s  current  acquisition  policy  and  guidance 
reflect  a  knowledge-based  acquisition  framework  in  line  with  acquisition  best 
practices  GAO  has  been  identifying  since  the  late  1990s.  We  will  use  this  framework 
in  this  report  to  analyze  both  the  VH-71  and  the  VXX  programs. 

A  knowledge-based  acquisition  framework  involves  achieving  the  right  knowledge  at 
the  right  time-enabling  leadership  to  make  informed  decisions  about  when  and  how 
best  to  move  into  various  acquisition  phases.  In  essence,  knowledge  supplants  risk 
over  time.  Our  best  practices  work  has  demonstrated  that  this  building  of  knowledge 
consists  of  information  that  should  be  gathered  at  three  critical  points  over  the 
course  of  a  program. 

•  Knowledge  point  1:  Resources  and  requirements  match.  Achieving  a  high 
level  of  technology  maturity  and  preliminary  system  design  backed  by  robust 
systems  engineering  by  the  start  of  system  development  is  an  important 
indicator  of  whether  this  match  has  been  made.  This  means  that  the 
technologies  needed  to  meet  essential  product  requirements  have  been 
demonstrated  to  work  in  their  intended  environment.  In  addition,  the 
developer  has  completed  a  preliminary  design  of  the  product  that  shows  the 
design  is  feasible.  If  the  above  conditions  are  met,  a  solid  business  case  is 
established  at  this  point. 

•  Knowledge  point  2:  Product  design  is  stable.  This  point  occurs  when  a 
program  determines  that  a  product’s  design  will  meet  customer  requirements, 
as  well  as  cost,  schedule,  and  reliability  targets.  A  best  practice  is  to  achieve 
design  stability  at  the  system-level  critical  design  review,  usually  held  midway 
through  system  development.  Completion  of  at  least  90  percent  of  engineering 
drawings  at  this  point  provides  tangible  evidence  that  the  product’s  design  is 
stable,  and  a  prototype  demonstration  shows  that  the  design  is  capable  of 
meeting  performance  requirements. 


^  In  December  2008,  DOD  revised  its  acquisition  instruction — Department  of  Defense  Instruction 
5000.02,  Operation  of  the  Defense  Acquisition  System.  The  Weapon  Systems  Acquisition  Reform  Act 
of  2009,  Pub.  L.  No.  111-23,  was  enacted  on  May  22,  2009. 
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•  Knowledge  point  3:  Mannfacturing  processes  are  mature.  This  point  is 
achieved  when  it  has  been  demonstrated  that  the  developer  can  manufacture 
the  product  within  cost,  schedule,  and  quality  targets.  A  best  practice  is  to 
ensure  that  all  critical  manufacturing  processes  are  in  statistical  control — that 
is,  they  are  repeatable,  sustainable,  and  capable  of  consistently  producing 
parts  within  the  product’s  quality  tolerances  and  standards — at  the  start  of 
production. 

Summary 

Several  lessons  learned  from  the  acquisition  strategy  and  eventual  termination  of  the 
VH-71  program  apply  to  the  VXX  program.  For  example,  had  the  VH-71  program 
followed  acquisition  best  practices  and  conducted  early  systems  engineering,  it  could 
have  led  to  a  feasible,  stable  preliminary  design  ideally  before  development  start.  In 
turn,  a  stable,  early  design  allows  for  more  accurate  program  cost  estimates  and  a 
better  foundation  for  sufficient  funding  commitments.  Instead,  it  began  without 
completing  systems  engineering  until  well  after  development  start.  As  a  consequence, 
it  never  achieved  design  stability  and  experienced  significant  cost  and  schedule 
problems  in  development.  Its  cost  estimates  doubled — from  about  $6.5  billion  at 
development  start  in  2005  to  about  $13  billion  when  terminated  in  2009.  More  than 
good  systems  engineering  is  necessary,  however.  A  key  to  successful  development  is 
the  ability  to  make  early  trade-offs  either  in  the  design  of  a  product  or  the  customer’s 
expectations  to  avoid  outstripping  the  resources  available  for  product  development. 
The  VH-71  program  was  not  afforded  room  needed  to  pursue  these  needed  trade-offs. 
Stringent  performance  requirements  (some  with  no  flexibility)  were  laid  out  for  the 
system  prior  to  the  start  of  development  and  did  not  appear  to  involve  significant 
consideration  of  trade-offs  of  cost,  performance,  and  schedule  negotiated  between 
the  customer  and  the  developer. 

The  VH-71  program’s  experience  validated  the  need  to  execute  a  knowledge-based 
acquisition  process  with  discipline,  confirming  the  danger  of  not  replacing  risk  with 
knowledge  earlier  in  the  acquisition  process.  VXX  program  officials  seem  to 
understand  this  lesson  learned  from  the  VH-71  program  and  appear  to  be  establishing 
a  knowledge-based  acquisition  process  emphasizing  early  systems  engineering.  One 
of  the  primary  lessons  they  learned  from  the  VH-71  program’s  experience  is  that  there 
must  be  an  early,  solid  business  case  with  a  rational  balance  between  requirements, 
cost,  and  schedule.  To  accomplish  this,  they  have  stated  that  a  rigorous  four-phase 
systems  engineering  and  technical  review  process  will  be  used.  Early  VXX  program 
efforts  appear  to  reflect  the  intent  to  pursue  a  knowledge-based  acquisition.  The  VXX 
program  is  currently  in  the  materiel  solution  analysis  phase  of  the  acquisition 
process,  and  an  ICD  has  been  developed  to  formally  document  the  capabilities 
required  to  perform  the  defined  mission,  the  specific  capability  gaps  that  exist,  and 
the  need  to  address  them.  Our  review  of  the  VXX  program’s  ICD  indicates  that  it 
addresses  all  three  of  these  areas  and  also  appears  to  align  with  acquisition  best 
practices.  An  AOA  has  been  initiated  but  not  yet  completed.  According  to  DOD 
officials,  it  will  be  more  robust  than  the  AOA  developed  for  the  VH-71  program.  The 
program  is  in  the  earliest  stages  of  development,  still  developing  a  business  case  to 
launch  product  development.  We  will,  as  you  have  requested,  assess  and  report  on  a 
wide  range  of  VXX  program  activities  moving  forward.  Throughout,  we  will  be 
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looking  for  the  implementation  of  a  knowledge-based  acquisition  through  application 
of  key  best  practice  process  controls. 

Lesson  Learned  from  the  VH-71  Program 

The  VH-71  program’s  failure  to  follow  acquisition  best  practices  was  a  critical  factor 
in  the  program’s  poor  performance  that  led  to  its  ultimate  termination.  It  started  with 
a  faulty  business  case,  did  not  perform  appropriate  systems  engineering  analysis  to 
gain  knowledge  at  the  right  times,  and  failed  to  make  necessary  trade-offs  between 
resources  and  requirements  even  after  years  of  development.  Because  of  these 
failures,  the  program  was  unable  to  achieve  stable  design  and  experienced  significant 
cost  growth  and  schedule  delays  before  being  terminated  in  2009. 

The  Program  Began  with  a  High-Risk  Business  Case 

The  VH-71  program  was  knowingly  initiated  with  a  high-risk  business  case.  To 
accelerate  the  program’s  initial  delivery  of  helicopters  from  2011  to  2008,  the  Navy 
adopted  a  two-step  acquisition  approach  and  initiated  production  at  the  same  time  it 
began  development.*  Prior  to  the  start  of  development,  a  March  2004  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(OUSD(AT&L))  Defense  Acquisition  Program  Support  Assessment  of  the  program 
concluded  that  the  program  had  a  high-risk  schedule  because  of  concurrent  design 
and  production  efforts.  However,  the  Office  of  the  Secretary  of  Defense  (OSD) 
recognized  and  accepted  the  program’s  risk  and  strategy  in  its  January  2005 
Acquisition  Decision  Memorandum  approving  the  program’s  entry  into  development. 
Concurrency  and  lack  of  systems  engineering  knowledge  to  achieve  a  match  between 
required  capabilities  and  resources  contributed  to  the  cost  overruns  and  schedule 
delays  experienced  by  the  program  prior  to  its  termination  in  2009. 

Systems  Engineering  Knowledge  Was  Not  Available 

The  risk  of  the  VH-71  program’s  business  case  was  even  higher  than  anticipated 
because  early  systems  engineering  wasn’t  done  when  it  should  have  been  to  be  in 
accord  with  acquisition  best  practices.  As  we  have  previously  reported,®  a  primary 
reason  for  cost  and  schedule  problems  is  too  many  technical  unknowns  and 
insufficient  knowledge  about  performance  and  production  risks.  If  this  early  systems 
engineering  is  not  performed,  significant  cost  increases  can  occur  as  the  system’s 
requirements  become  better  understood  by  the  government  and  contractor.  The  VH- 
71  program’s  experience  exemplified  this.  System  requirement  reviews  are  conducted 
to  ensure  that  system  requirements  have  been  properly  identified  and  that  a  mutual 


If  done  correctly,  the  use  of  an  incremental  approach  to  development  is  a  best  practice  that  our  work 
has  shown  enables  organizations  to  achieve  a  match  between  needs  and  resources.  Under  this 
approach,  basic  requirements  are  achieved  first,  with  additional  capabilities  planned  for  future 
generations  of  the  product.  Because  product  development  is  incremental,  achieving  knowledge  is 
more  manageable.  As  we  have  reported,  commercial  companies  have  found  that  trying  to  capture  the 
knowledge  needed  to  stabilize  the  design  of  a  product  with  considerable  new  technical  content  is  an 
unwieldy  task — especially  if  the  goal  is  to  reduce  development  cycle  times  and  get  the  product  to  the 
marketplace  as  quickly  as  possible.  GAO,  Best  Practices:  Using  a  Knowledge-Based  Approach  to 
Improve  Weapon  Acquisition,  GAO-04-386SP  (Washington,  D.C.:  January  2004). 

^  GAO,  Defense  Acquisitions:  Strong  Leadership  Is  Key  to  Planning  and  Executing  Stable  Weapon 
Programs,  GAO-10-522  (Washington,  D.C.:  May  6,  2010). 
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understanding  between  the  government  and  the  contractor  exists.  These  steps  should 
occur  prior  to  the  start  of  development.  However,  when  the  VH-71  program’s  system 
requirements  review  occurred  in  May  2005 — 4  months  after  the  start  of 
development — it  was  found  that  the  contractor’s  design  was  significantly 
noncompliant  with  what  the  contract  required.  In  June  2006,  the  VH-71  program’s 
Defense  Acquisition  Executive  Summary  reported  that  it  had  taken  considerable 
effort  over  the  prior  14  months  to  resolve  related  issues  and  that  good  progress  had 
been  made  in  aligning  the  contractor’s  design  with  what  the  contract  required.  It  also 
noted,  though,  that  related  activities  had  resulted  in  delay  of  technical  reviews  and 
contract  milestones  and  additional  cost. 

Having  a  feasible,  stable  preliminary  design  for  a  weapons  program  early  in  the 
acquisition  process  is  important  for  lessening  risk,  both  by  ensuring  that  there  is  a 
match  between  resources  and  requirements  and  by  demonstrating  that  a  product’s 
design  can  meet  customer  requirements,  as  well  as  cost,  schedule,  and  reliability 
targets.  The  VH-71  program  did  not,  however,  complete  needed  systems  engineering 
until  well  after  the  start  of  development  for  both  increments  and,  in  the  case  of 
Increment  1,  after  approval  of  production  as  well,  as  Increment  1  was  approved  to 
enter  development  and  production  at  the  same  time.  OSD  approved  the  Increment  1 
entry  into  development  and  production  in  January  2005.  A  preliminary  design  review 
(PDR),  which  best  practices  indicate  should  occur  prior  to  a  program  entering 
development,  was  not  completed  for  the  Increment  1  effort  until  February  2006 — 13 
months  after  the  start  of  development.  OSD  also  approved  Increment  2  development 
in  January  2005.  A  PDR  for  Increment  2  had  not  occurred  by  the  time  a  stop  work 
order  was  placed  on  the  Increment  2  effort  in  December  2007 — 35  months  after  the 
start  of  Increment  2  development.  The  stop  work  order  was  placed  as  a  result  of  a 
concern  that  the  required  work  would  exceed  available  funding. 

Similarly,  it  is  a  best  practice  to  achieve  design  stability  at  a  system-level  critical 
design  review  (CDR),  usually  held  midway  through  system  development — that  is, 
halfway  between  the  start  of  development  and  approval  to  enter  production.  A  CDR 
for  Increment  1  was  held  in  August  2006  but  was  not  completed  as  significant 
elements  remained  to  be  addressed.  Subsequently,  an  airframe  CDR  did  not  occur 
until  February  2007 — more  than  2  years  after  the  start  of  Increment  1  development 
and  production.  A  CDR  was  scheduled  for  Increment  2  to  occur  in  fiscal  year  2008, 
but  the  stop  work  order  for  Increment  2  was  issued  prior  to  the  scheduled 
completion  of  the  CDR.  As  knowledge  from  the  successive  systems  engineering 
reviews  builds,  uncertainty  and  associated  risks  in  the  cost  estimate  decrease.  As 
noted,  this  knowledge  was  developed  late,  if  at  all,  in  the  VH-71  program  and  the 
consequence  was  cost  growth  to  the  degree  that  at  the  time  the  VH-71  program  was 
terminated  in  June  2009,  it  had  grown  from  a  total  estimated  cost  of  about  $6.5  billion 
at  the  start  of  development  in  January  2005  to  about  $13  billion. 

The  Program  Did  Not  Make  Appropriate  Trade-offs  between  Capability  Requirements 
and  Allowable  Resources 


While  good  systems  engineering  can  identify  and  inform  trade-offs,  the  customer  and 
developer  must  be  willing  to  make  trade-offs  to  achieve  a  successful  business  case. 
We  have  found  that  key  to  successful  developments  was  the  ability  to  make  early 
trade-offs  either  in  the  design  of  the  product  or  the  customer’s  expectations  to  avoid 


Page  6 


GAO-11-380R  Presidential  Helicopter  Program 


outstripping  the  resources  available  for  product  development.  Conversely,  as  we  have 
found  with  other  programs — such  as  the  Armed  Reconnaissance  Helicopter 
program — an  unwillingness  to  make  performance  trade-offs  can  contribute  to 
programs  being  unexecutable,  ultimately  resulting  in  their  termination. 

The  VH-71  program  was  not  afforded  sufficient  room  needed  to  pursue  needed  trade¬ 
offs.  Stringent  performance  requirements  (some  with  no  flexibility)  were  laid  out  for 
the  system  prior  to  the  start  of  development  and  did  not  appear  to  involve  significant 
consideration  of  trade-offs  of  cost,  performance,  and  schedule  negotiated  between 
the  customer  and  the  developer.  Rather,  as  a  July  2007  Institute  for  Defense 
Analyses  independent  review  team  reported  on  the  executability  of  Increment  2, 

“Unfortunately,  the  requirements  for  the  program  are  still  not  well 
founded.  Initially  this  reflected  a  hurried  program  start,  without  the 
foundation-laying  analyses  and  design  and  requirements  tradeoffs, 
necessary  to  initiate  a  program  with  an  acceptable  level  of  stability. 

Currently,  incomplete  requirements  documentation  and  poor 
configuration  management  presage  further  instability  in  the  future.  The 
[independent  review  team]  also  observed  that  the  two  directions  given  to 
the  program  by  the  Acquisition  Decision  Memorandum  were  not  followed: 

•  Leadership  engagement  and  clear  guidance  were  imperative  for 
program  success. 

•  Disciplined  application  of  systems  engineering  practices,  early 
configuration  control  decisions,  and  strict  configuration  control 
were  essential.” 

VXX  Program  Appears  Set  to  Use  a  Knowledge-Based  Acquisition  Approach 

The  VH-71  program  experience  validated  the  need  to  execute  a  knowledge-based 
process  with  discipline  and  confirmed  the  danger  of  not  replacing  risk  with 
knowledge  earlier  in  the  acquisition  process.  VXX  program  officials  seem  to 
understand  this  and  appear  to  be  moving  forward  in  an  attempt  to  apply  these 
lessons.  They  have  stated  that  their  aim  is  to  establish  an  initial  knowledge-based 
acquisition  business  case  that  will  emphasize  early  systems  engineering;  mature 
technologies;  an  incremental,  knowledge-based  approach;  and  the  ability  to  trade 
capability  for  resources.  The  program  is  currently  in  its  earliest  stages,  nearing 
completion  of  the  materiel  solution  analysis  phase,  and  is  likely  still  years  away  from 
establishing  its  business  case.  Therefore,  we  do  not  have  much  acquisition 
documentation  to  review  beyond  the  ICD,  information  on  the  preliminary  acquisition 
planning,  and  AOA  plans. 

VXX  program  management  indicated  that  one  of  the  primary  lessons  learned  from  the 
VH-71  program  experience  is  that  DOD  must  establish  and  maintain  a  solid  business 
case,  where  a  balance  exists  between  requirements,  cost,  and  schedule  that  results  in 
an  executable  program  with  acceptable  risk.  To  accomplish  this,  the  program  intends 
to  use  a  rigorous  four-phase  systems  engineering  and  technical  review  process 


“  Institute  for  Defense  Analyses,  VH-  71  Presidential  Helicopter  Program  Assessment,  IDA  Paper  P- 
4243  (Alexandria,  Va.,  July  2007)  (Distribution  authorized  to  U.S.  government  agencies  and  their 
contractors;  Specific  Authority.  Other  requests  must  be  referred  to  OUSD(AT&L).) 
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constituting  a  more  knowledge-based  acquisition  approach.  This  process  differs 
greatly  from  the  process  followed  during  the  VH-71  program.  According  to  VXX 
program  officials,  the  VXX  program’s  systems  engineering  and  technical  review 
process  will  begin  earlier  during  the  technology  development  phase  (pre-Milestone 
B).  VXX  program  officials  stated  that  they  plan  to  enter  the  systems  development 
phase  with  a  preliminary  technical  design  and  possibly  an  early  prototype  of 
subsystems. 

Early  VXX  program  efforts  appear  to  reflect  the  intent  to  pursue  a  best  practices- 
aligned  knowledge-based  acquisition.  The  VXX  program  is  currently  in  the  pre- 
Milestone  A“  phase  of  the  acquisition  process.  An  ICD  has  been  approved  and  an 
AOA  is  currently  being  finalized;  both  are  required  for  Milestone  A.  After  termination 
of  the  VH-71  program  contract,  DOD  conducted  an  analysis  to  determine  if  a 
nonmateriel  solution  existed  to  fill  the  identified  gap  in  the  presidential  helicopter 
program’s  vertical  lift  capability.  DOD  determined  that  there  were  no  nonmaterial 
approaches  that  would  significantly  improve  or  resolve  the  capability  gaps.^^  It  was 
determined  that  a  materiel  solution  must  be  pursued,  and  as  a  result,  an  ICD  for  the 
VXX  program  was  developed  in  August  2009.  According  to  the  Chairman  of  the  Joint 
Chiefs  of  Staff  instruction  that  guides  the  process  that  develops  ICDs,  an  ICD  is  a 
formal  capability  assessment  by  a  military  service,  defense  agency,  or  other  sponsor 
and  it  formally  documents  (1)  the  capabilities  required  to  perform  the  defined 
mission,  (2)  the  specific  capability  gap  or  set  of  gaps  that  exist,  and  (3)  the  need  to 
address  the  capability  gap(s).^^  Our  review  of  the  VXX  program’s  ICD  indicates  that  it 
addressed  all  three  of  these  areas.  The  ICD  and  its  intended  use  also  appear  to  align 
with  acquisition  best  practices.  Unlike  the  requirements  document  that  led  to  the  VH- 
71  program,  the  VXX  program’s  ICD  identifies  performance  attributes  as  “preferred 
values”  rather  than  key  performance  parameters.  Program  officials  indicated  that  the 
use  of  preferred  values  instead  of  firm  requirements  will  allow  the  program  room  to 
do  trade-offs  needed  to  achieve  a  match  between  the  users’  requirements  and 
available  resources  before  development  begins — an  acquisition  best  practice. 

According  to  a  DOD  official,  the  Navy  expects  to  issue  the  results  of  its  AOA  in  the 
second  quarter  of  2011.  According  to  DOD  officials,  it  will  be  more  robust  than  the 
AOA  used  to  support  the  VH-71  program.  While  we  cannot  assess  its  sufficiency  until 
it  is  complete,  statements  by  DOD  officials  on  the  nature  of  the  AOA  effort  suggest 
the  pursuit  of  an  acquisition  approach  in  line  with  best  practices.  Plans  are  for  the 
AOA  to  analyze  in  detail  the  estimated  cost  and  effectiveness  of  a  range  of  potential 
materiel  solutions  to  support  the  need.  In  the  past,  we  have  reported  that  many  AO  As 
do  not  effectively  consider  a  broad  range  of  alternatives  for  addressing  a  need  or 
assess  technical  and  other  risks  associated  with  each  alternative.^^  Without  a 
sufficient  comparison  of  alternatives  and  focus  on  technical  and  other  risks,  AOAs 


“  The  VXX  Milestone  A  is  currently  expected  to  occur  sometime  in  fiscal  year  2011. 

The  analysis  typically  evaluates  military  doctrine,  organization,  training,  materiel,  leadership  and 
education,  personnel,  and  facilities  to  try  to  identify  a  nonmaterial  solution  to  a  capability  gap  or  gaps. 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170. OIG,  Joint  Capabilities  Integration  and 
Development  System  (2009),  end.  A,  2(d).  When  the  Joint  Requirements  Oversight  Council  approves 
an  ICD,  it  validates  the  capabilities  required  to  perform  the  mission  as  defined,  the  gaps  in  capabilities 
along  with  their  priorities  and  operational  risks,  and  the  need  to  address  the  capability  gaps. 

GAO,  Defense  Acquisitions:  Many  Analyses  of  Alternatives  Have  Not  Provided  a  Robust 
Assessment  of  Weapon  System  Options,  GAO-09-665  (Washington,  D.C.:  Sept.  24,  2009). 
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may  identify  solutions  that  are  not  feasible  and  decision  makers  may  approve 
programs  based  on  limited  knowledge.  While  many  factors  can  affect  cost  and 
schedule  outcomes,  we  have  found  that  programs  that  had  a  limited  assessment  of 
alternatives  tended  to  have  poorer  outcomes  than  those  that  had  more  robust  AO  As. 
When  the  AOA  is  issued,  we  will  assess  it  for  robustness — the  range  of  alternatives  it 
considers,  its  depth  of  analysis,  and  its  consideration  of  trade-offs. 

As  the  program  progresses,  we  will  continue  to  assess  and  report  its  performance  as 
directed  in  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year  2011 
(the  Act).  Figure  1  illustrates  the  acquisition  process  and  where  the  VXX  program 
currently  stands  in  that  process,  and  identifies  some  of  the  key  program  documents 
we  will  review  in  the  future  and  the  best  practices  criteria  that  the  program  should 
meet  as  it  progresses.  We  will  use  this  as  a  template  for  future  reviews  of  the 
program. 

Figure  1 :  Alignment  of  DOD’s  Acquisition  Process,  Key  Program  Documents,  and  Best  Practices 
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Key  steps: 

•  PDR  completed 

•  Technologies  demonstrated  to  high 
levels 

•  Incremental  acquisition  strategy  in 
place 

•  Knowledge-based  cost  estimate 


Knowledge  Point  2 

Design  is  stable  and  performs 

as  expected. 

Decision  to  start  building  and  testing 

production  representative  prototypes. 

Key  steps: 

•  System-level  CDR  and  subsystem 
design  reviews  completed 

•  Ninety  percent  of  engineering 
drawings  released 

•  Integrated  system  prototype 
demonstrated 

•  Critical  manufacturing  processes 
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Key  steps: 

•  Production-representative 
prototype  demonstrated  in  intended 
environment 
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•  Product  reliability  demonstrated  via 
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Source;  GAO  analysis  of  DOD's  acquisition  process  and  GAO's  best  practices. 
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By  using  this  framework,  we  will  address  the  issues  identified  in  the  Act.  Specifically, 

•  the  extent  to  which  the  program  is  meeting  development  and  procurement 
cost,  schedule,  performance,  and  risk  mitigation  goals; 

•  the  progress  of  developmental  and  operational  testing  of  aircraft  and  plans  for 
correcting  deficiencies  in  aircraft  performance,  operational  effectiveness, 
reliability,  suitability,  and  safety; 

•  the  program  procurement  plans,  production  results,  and  efforts  to  improve 
manufacturing  efficiency  and  supplier  performance; 

•  the  program’s  acquisition  strategy,  including  whether  it  is  in  accord  with 
acquisition  best  practices  and  DOD’s  acquisition  policy  and  regulations; 

•  risk  assessments  of  its  integrated  master  schedule  and  test  and  evaluation 
master  plan;  and 

•  our  assessment  of  the  sufficiency  of  the  program’s  ICD  (if  revised),  AOA, 
capabilities  development  document,  and  a  systems  requirement  document  (if 
produced). 

Overall,  we  will  assess  the  use  of  acquisition  best  practices  throughout  the  VXX 
program’s  acquisition  through  management’s  adherence  to  key  best  practice  model 
controls.  Those  controls  are  outlined  in  the  table  in  enclosure  I. 

Agency  Comments 

DOD  provided  technical  comments  on  the  information  in  this  report,  which  GAO 
incorporated  as  appropriate,  but  declined  to  provide  additional  comments. 


We  are  sending  copies  of  this  report  to  interested  congressional  committees;  the 
Secretary  of  Defense;  the  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics;  and  the  Secretary  of  the  Navy.  This  report  also  is  available  at  no  charge 
on  GAO’s  Web  site  at  http://www.gao.gov. 

Should  you  or  your  staff  have  any  questions  on  the  matters  covered  in  this  report, 
please  contact  me  at  (202)  512-4841  or  .sullivanm@gao.  gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Key  contributors  to  this  report  are  listed  in  enclosure  II. 


Michael  J.  Sullivan,  Director 
Acquisition  and  Sourcing  Management 

Enclosures  -  2 
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List  of  Committees 


The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Howard  P.  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  C.  W.  Bill  Young 
Chairman 

The  Honorable  Norman  D.  Dicks 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Enclosure  I:  Best  Practices  Model  Controls  at  Key  Acquisition  Process 

Points 


Criteria 

Milestone  A:  Occurs  as  programs  begin  the 
technology  development  phase. 

•  Complete  robust  Analysis  of  Alternatives. 

Milestone  B  (knowledge  point  1):  Occurs  as 
programs  begin  the  engineering  and 
manufacturing  development  phase  (Milestone  B). 
Match  exists  between  requirements  and 
resources.  Completed  when  technologies 
needed  to  meet  essential  product  requirements 
have  been  demonstrated  to  work  in  their 
intended  environments  and  the  producer  has 
completed  a  preliminary  design  of  the  product. 

•  Demonstrate  high  technology  readiness  levels. 

•  Ensure  that  product  requirements  are  informed  by 
the  systems  engineering  process. 

•  Establish  cost  and  schedule  estimates  for  the 
product  based  on  knowledge  from  preliminary 
design  using  systems  engineering  tools. 

•  Complete  preliminary  design  review. 

•  Conduct  decision  review  for  program  launch. 

Critical  design  review  (knowledge  point  2): 

Occurs  at  the  critical  design  review  between 
integration  and  demonstration.  Completed  when 
design  is  stable  and  has  been  demonstrated 
through  prototype  testing  and  90  percent  of 
engineering  drawings  are  releasable  to 
manufacturing  organizations. 

•  Complete  90  percent  of  design  drawings. 

•  Complete  subsystem  and  system  design  reviews. 

•  Demonstrate  with  prototype  that  design  meets 
requirements. 

•  Obtain  stakeholder  concurrence  that  drawings  are 
complete  and  producible. 

•  Complete  failure  modes  and  effects  analysis. 

•  Identify  key  system  characteristics. 

•  Identify  critical  manufacturing  processes. 

•  Establish  reliability  targets  and  growth  plan  based 
on  demonstrated  reliability  rates  of  components 
and  subsystems. 

•  Conduct  design  review  to  enter  system 
demonstration. 

Milestone  C  (knowledge  point  3):  Occurs  at 
low-rate  initial  production  commitment. 

Completed  when  product  is  ready  to  be 
manufactured  within  cost,  schedule,  and  quality 
targets  and  all  key  manufacturing  processes  are 
under  statistical  control  and  product  reliability 
has  been  demonstrated. 

•  Demonstrate  manufacturing  processes. 

•  Build  production-representative  prototypes. 

•  Test  production-representative  prototypes  to 
achieve  reliability  goal. 

•  Test  production-representative  prototypes  to 
demonstrate  the  product  in  a  realistic  environment. 

•  Collect  statistical  process  control  data. 

•  Demonstrate  that  critical  processes  are  capable 
and  under  statistical  control. 

•  Conduct  decision  review  to  begin  production. 

Source:  GAO. 
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